Method of generating features optimal to a dataset and classifier

ABSTRACT

A method of generating features optimal to a particular dataset and classifier is disclosed. A dataset of messages is inputted and a classifier is selected. An algebra of features is encoded. Computable features that are capable of describing the dataset from the algebra of features are selected. Irredundant features that are optimal for the classifier and the dataset are selected.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

The invention was made with Government support under ContractDE-AC05-76RLO1830, awarded by the U.S. Department of Energy. TheGovernment has certain rights in the invention.

TECHNICAL FIELD

This invention relates to machine learning. More specifically, thisinvention relates to a method of producing an optimal set of featuresfor a given dataset and classifier.

BACKGROUND OF THE INVENTION

In machine learning, users are often interested in understanding how toclassify data given examples of previously classified data (a trainingset). Many algorithms have been devised to tackle this problem and eachhas its strengths and weaknesses. Nonetheless, many if not all machinelearning systems process features rather than the raw data. For example,a piece of fruit can be classified as an orange or an apple. A salientfeature for classification might be the color of the fruit.

Despite the importance of features in classification, most algorithmsrequire that the user specify the features that should be utilized.While this may not appear to be an obstacle, people are typically bad atselecting important irredundant features from large complex data sets.Similarly, feature selection (removal of redundant features) istypically accomplished after feature generation, not in tandem with it.Furthermore, human selection is time consuming and very subjective—oftenperformed by selecting features that a human may use to performclassification rather than what might be useful to the underlyingclassification algorithm.

SUMMARY OF THE INVENTION

The present invention is directed to methods of generating featuresoptimal to a particular dataset and classifier. In one embodiment, themethod includes inputting a dataset of messages; inputting a classifier;encoding an algebra of computable features capable of describing thedata set; selecting computable irredundant features from the algebra offeatures which are optimal for the classifier and the dataset.

In one embodiment, the method further comprises randomly selecting afirst subset of the computable features and inputting them into theclassifier.

In one embodiment, the method further comprises determining accuracy ofthe classifier using a validation set and each feature independently.

In one embodiment, the method further comprises randomly selecting asecond subset of features from the first subset of computable features.

In one embodiment, the method further comprises mutating the secondsubset of features.

In one embodiment, the method further comprises randomly selecting pairsof features from the first subset of computable features.

In one embodiment, the method further comprises applying crossing toeach selected pair of features from the first subset of computablefeatures.

In one embodiment, the method further comprises combining the firstsubset of computable features, the mutated features, and the crossedfeatures to generate a third subset of features.

In one embodiment, the method further comprises selecting a fourthsubset of features from the algebra of features and adding to the thirdsubset of features.

In one embodiment, the method further comprises removing duplicatefeatures from the third subset of features.

In one embodiment, the method further comprises inputting the featuresfrom the third subset into the classifier and computing accuracy of theclassifier for each feature using a validation set.

In one embodiment, the method further comprises ranking the features ofthe third subset by their corresponding classifier accuracy.

In one embodiment, the method further comprises maintaining only fittestfeatures from the third subset to form a fifth subset of the same sizeas the first subset, and replacing the first subset with the fifthsubset.

In one embodiment, the method further comprises repeating the mutating,the crossing, the removing, the ranking, the selecting, and themaintaining until accuracy of a most fit feature converges, wherein themost fit feature is cached.

In another embodiment of the present invention, a method of generatingirredundant features optimal to a particular dataset and classifier isdisclosed. The method includes inputting a dataset of messages;inputting a classifier; encoding an algebra of computable featurescapable of describing the dataset through expression trees; and applyinga genetic algorithm to select a resulting feature which maximizesaccuracy of the classifier given a validation set.

In one embodiment of the present invention, the features are encoded asgrammar trees utilizing a BNF grammar or a Chomsky grammar.

In one embodiment, the resulting feature is cached and the algorithmiteratively reapplied to determine a set of irredundant featuresmaximizing classifier accuracy for the dataset. The classifier utilizesthe cached feature and a candidate feature when computing accuracy

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a method of generating features optimal to aparticular dataset and classifier, in accordance with one embodiment ofthe present invention.

FIG. 2 is a flowchart of a method of generating features optimal to aparticular dataset and classifier, in accordance with one embodiment ofthe present invention.

FIG. 3A shows a first subset of computable features using threeexpression trees.

FIG. 3B shows a ranking of the features which corresponds to accuracy ofthe selected classifier.

FIG. 3C shows a mutation step multiplying the weight and height, whichchanges multiplication to addition.

FIG. 3D shows a crossing step applied to a selected pair of features.

FIG. 3E shows a third subset of features comprising the first subset ofcomputable features (from FIG. 3A), the mutated features (from FIG. 3C),and the crossed features (from FIG. 3D).

FIG. 3F shows the third subset of features after adding a randomlyselected computable feature, the fourth subset, to the third subset offeatures.

FIG. 3G shows a ranking of the third subset of features by theircorresponding classifier accuracy.

FIG. 3H shows a fifth subset of features which comprises only the threefittest features from the third subset.

FIG. 4A shows two 3×3 matrices of pixel representations for each letter.

FIG. 4B shows a first subset of computable features using two trees andmatrices.

FIG. 4C shows a ranking of the features which correspond to accuracy ofthe selected classifier.

FIG. 4D shows a mutation step that changes a pixel from each matrix.

FIG. 4E shows a crossing step applied to the matrices.

FIG. 4F shows a third subset of features comprising the first subset offeatures, the mutated features, and the crossed features.

FIG. 4G shows the third subset of features after adding a randomlyselected computable feature, the fourth subset, to the third subset offeatures.

FIG. 4H shows a ranking of the third subset of features by theircorresponding classifier accuracy.

FIG. 4I shows a fifth subset of features which comprises only the twofittest features from the third subset.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is directed to methods of generating featuresoptimal to a particular dataset and classifier by iteratively applyinggenetic algorithms. The present invention improves the accuracy ofstandard machine learning algorithms by extracting salient features fromthe data and selecting irredundant features. The present inventionallows a user to select a classifier or classification scheme and a setof fixed input features that must be used when determining classifieraccuracy.

In one embodiment, elements of this algebra of features are encodedthrough grammar trees specified by a BNF grammar or a Chomsky grammar.The trees are uniformly randomly generated and a genetic algorithm isapplied to select a feature which maximizes the accuracy of a userspecified classifier given a training set and a validation set. Theresulting feature is cached and the algorithm is reapplied to determinethe most efficient feature given the features determined in the earliersteps. Given sufficient time, the algorithm will always converge andproduce the optimal set of features for a given dataset and classifier.

FIG. 1 is a flowchart 100 of a method of generating features optimal toa particular dataset and classifier, in accordance with one embodimentof the present invention. At 110, a dataset of messages is inputted. Themessages are constructed from discrete values such as integers and/orletters. At 120, a classifier is inputted. The classifier may be a userselected classifier or classification scheme. At 130, an algebra offeatures is encoded. This algebra is used mathematically to add,subtract and/or multiply features and multiply by coefficients. At 140,computable features are selected that are capable of describing thedataset from the algebra of features. The computable features includethe use of coefficients that must be computable numbers, such as finiteprecision numbers, and also refers to the depth of grammar trees. Thefeatures selected are non-constant on the dataset of messages. At 150,irredundant features are selected that are optimal for the classifierand the dataset.

The method may further include randomly selecting a first subset of thecomputable features and inputting them into the classifier.

The accuracy of the classifier may be determined using a validation set.The inputted dataset can be split into a validation set and a trainingset.

The method may further include randomly selecting a second subset offeatures from the first subset of computable features. This selection isperformed randomly with a features probability being proportional to theaccuracy of the corresponding classifier.

The second subset of features may be mutated in an attempt to improvefitness of the features. For example, given a feature encoded as anexpression tree, a node in the tree may be picked and replaced with acompatible node in accordance with an underlying grammar.

The method may also include randomly selecting pairs of features fromthe first subset of computable features and applying crossing to eachselected pair of features. This selection process is also proportionalto the feature accuracy. As an example, a node from one feature and anode from another feature are selected such that the nodes arecompatible, and the two sub-trees rooted at the selected nodes areswapped in an attempt to improve feature fitness. The nodes are selectedin accordance with an underlying grammar to ensure that the resultingtrees form valid expressions.

The method may further include combining the first subset of computablefeatures, the mutated features, and the crossed features to generate athird subset of features. Duplicate features are removed from the thirdsubset of features. This removing step is performed, for example, toremove a feature that is the same as another feature in the set.

The features from the third subset may be inputted into the classifier,wherein the accuracy of the classifier is computed. This allows thefeatures of the third subset to be ranked by their correspondingclassifier accuracy. For example, a list of features may include afeature at the top of the list that is the “most fit” feature, thesecond to the top feature that is the next “most fit” feature, and soon. Any ties in the ranking are broken based on a total order so thatthere is no ambiguity left. For example, ties can be broken between twointegers if one is less than the other.

The method may further include maintaining only the fittest featuresfrom the third subset to form a fourth subset of the same size as thefirst subset, and replacing the first subset with the fourth subset.

The previous steps above—the mutating, the crossing, the removing, theranking, and the maintaining steps—maybe repeated until accuracy of themost fit feature converges. The most fit feature is added to a fifthsubset of features, which was constructed to use in the iterativeprocess of constructing irredundant features. These iterations occuruntil the most fit feature for a selected classifier is determined.

FIG. 2 is a flowchart of a method of generating features optimal to aparticular dataset and classifier, in accordance with one embodiment ofthe present invention. At 210, a dataset of messages is inputted. At220, a classifier is inputted. At 230, an algebra of computable featurescapable of describing the dataset through expression trees is encoded.The trees may be specified by a BNF grammar or a Chomsky grammer. At240, a genetic algorithm is applied for selecting a resulting featurewhich maximizes accuracy of the classifier given a validation set.

Example 1

The following examples serve to illustrate certain embodiments andaspects of the present invention and not to be construed as limiting thescope thereof.

As one example, the present invention can be used to determine if aperson is obese—or classified as obese or not obese. The dataset caninclude features extracted from medical records. These features caninclude, but not limited to, height, weight, ape index, ethnicity, haircolor, and age. Alternatively, a user can select which features to usethat are relevant to obesity. The user also selects a classifier.

Expression trees specified by a BNF grammar are used to describe andcombine the features. FIG. 3A illustrates a first subset of computablefeatures using three expression trees. The first expression tree on theleft adds ape index to age. The second expression tree in the middletakes weight and divides by height. The third expression tree on theright multiplies weight by height.

FIG. 3B shows a ranking of the features which corresponds to accuracy ofthe selected classifier. The expression tree on the left—weight dividedby height—was highest ranked or most relevant to obesity from the firstsubset of features, while the expression tree on the right—ape indexadded to age—was the least useful for obesity from the same subset.

A second subset from the first subset of features is selected. In thisexample, the features of the expression tree multiplying the weight andheight is randomly selected and then mutated. The mutation processchanges multiplication to addition. This is shown in FIG. 3C.

Pairs of features from the first subset of computable features (fromFIG. 3A) are selected, and crossing is applied to the selected pair offeatures. The selected pairs are shown on the left of FIG. 3D, and theresulting pair from the crossing step—where nodes of the trees areswapped—is shown on the right of FIG. 3D.

FIG. 3E shows a third subset of features comprising the first subset ofcomputable features (from FIG. 3A), the mutated features (from FIG. 3C),and the crossed features (from FIG. 3D).

FIG. 3F shows the third subset of features after adding a randomlyselected computable feature, the fourth subset, to the third subset offeatures.

The third subset of features is ranked by their corresponding classifieraccuracy. The ranking from best to worst—left to right—is shown in FIG.3G.

Duplicate features, if any, are removed from the third subset offeatures. In this manner, ties are broken based on a total order. Forexample, the two left most trees—w/(2 h) and w/h—are compared. Since 2is less than h in our total order, the ties are resolved in favor of theleft most tree (w/2 h).

A fifth subset is shown in FIG. 3H. This fifth subset is the same sizeas the first subset (three trees as shown in FIG. 3A) and comprises onlythe three fittest features from the third subset.

The process is repeated until accuracy of a most fit feature converges,which is then cached. This results in the most fit feature—w/h²—fordetermining obesity, which happens to be BMI.

Example 2

As another example, the present invention can be used for opticalcharacter recognition (e.g. handwriting recognition). FIG. 4A shows two3×3 matrices of pixel representations for the letter L and the letter n.The pixels of the two binary images are 0's and 1's. This means thateach pixel in the matrix is stored as a single bit—i.e., a 0 or a 1. Thetwo matrices correlate to the dataset of messages for this example.

Trees which encode matrices are used to describe and combine thefeatures. FIG. 4B illustrates a first subset of computable featuresconsisting of two expression trees depicted with their correspondingmatrices. The first tree and matrix on the left includes two Os or blackpixels and two is or white pixels. The second tree and matrix on theright also includes two 0 pixels and two 1 pixels but in a differentconfiguration or order.

FIG. 4C shows a ranking of the features which correspond to accuracy ofthe selected classifier. The tree and the matrix on the left—with pixelorder of 0011—was highest ranked or more relevant to characterrecognition from the first subset of features, while the tree and matrixon the right was less useful for character recognition from the samesubset.

The rankings are determined from the convolution. In this case theresults of the two matrices in FIG. 4B are the same (i.e., they aretied). The order is then chosen based on a total order. In thiscase—reading the tree along the bottom—0011 is lower in dictionary orderthan 0110 and so it comes first.

The mutation changes the first pixel of the first tree from 0 to 1, asshown in FIG. 4D. Crossing is applied to the first subset of features.The second pixel of each tree or matrix is swapped, as shown in FIG. 4E.

FIG. 4F shows a third subset of features comprising the first subset offeatures (from FIG. 4B), the mutated features (from FIG. 4D), and thecrossed features (from FIG. 4E).

FIG. 4G shows the third subset of features after adding a randomlyselected computable feature, the fourth subset, to the third subset offeatures.

The third subset of features is ranked by their corresponding classifieraccuracy. The ranking from best to work—left to right—is shown in FIG.4H. Duplicate features, if any, are removed from the third subset offeatures.

A fifth subset is shown in FIG. 4I. This fifth subset is the same sizeas the first subset (two trees/matrices as shown in FIG. 4B) andcomprises only the two fittest features from the third subset. As can benoticed from the matrices of FIG. 4I, the matrix on the left ‘1011’resembles the letter ‘L’ more than the matrix on the right with pixelorder ‘0011’.

The process can be repeated until accuracy of a most fit featureconverges, which is then cached.

The present invention has been described in terms of specificembodiments incorporating details to facilitate the understanding of theprinciples of construction and operation of the invention. As such,references herein to specific embodiments and details thereof are notintended to limit the scope of the claims appended hereto. It will beapparent to those skilled in the art that modifications can be made inthe embodiments chosen for illustration without departing from thespirit and scope of the invention.

We claim:
 1. A method of generating features optimal to a particulardataset and classifier, comprising: a. inputting a dataset of messages;b. inputting a classifier; c. encoding an algebra of features; d.selecting computable features capable of describing the dataset from thealgebra of features; and e. selecting an optimal feature that is optimalfor the classifier and the dataset.
 2. The method of claim 1 furthercomprising randomly selecting a first subset of the computable featuresand inputting them into the classifier.
 3. The method of claim 2 furthercomprising determining accuracy of the classifier using a validationset.
 4. The method of claim 3 further comprising randomly selecting asecond subset of features from the first subset of computable features.5. The method of claim 4 further comprising mutating the second subsetof features.
 6. The method of claim 5 further comprising randomlyselecting pairs of features from the first subset of computablefeatures.
 7. The method of claim 6 further comprising applying crossingto each selected pair of features from the first subset of computablefeatures.
 8. The method of claim 7 further comprising combining thefirst subset of computable features, the mutated features, and thecrossed features to generate a third subset of features.
 9. The methodof claim 8 further comprising selecting a fourth subset of features fromthe algebra of features and adding the fourth subset of features to thethird subset of features.
 10. The method of claim 9 further comprisingremoving duplicate features from the third subset of features.
 11. Themethod of claim 10 further comprising inputting the features from thethird subset into the classifier and computing accuracy of theclassifier.
 12. The method of claim 11 further comprising ranking thefeatures of the third subset by their corresponding classifier accuracy.13. The method of claim 12 further comprising breaking ties, if any, inthe ranking based on a total order.
 14. The method of claim 13 furthercomprising maintaining only fittest features from the third subset toform a fifth subset of the same size as the first subset, and replacingthe first subset with the fifth subset.
 15. The method of claim 14further comprising repeating the mutating, the crossing, the removing,the ranking, the selecting, and the maintaining until accuracy of a mostfit feature converges, wherein the most fit feature is cached.
 16. Amethod of generating irredundant features optimal to a particulardataset and classifier, comprising: a. inputting a dataset of messages;b. inputting a classifier; c. encoding an algebra of computable featurescapable of describing the dataset through expression trees; and d.applying a genetic algorithm to select a resulting feature whichmaximizes accuracy of the classifier given a validation set.
 17. Themethod of claim 16 wherein the resulting feature is cached and thealgorithm iteratively reapplied to determine a set of irredundantfeatures maximizing classifier accuracy for the dataset, wherein whencomputing accuracy the classifier utilizes the cached features and acandidate feature.
 18. The method of claim 16 wherein the trees arespecified by a BNF grammar or a Chomsky grammar.
 19. The method of claim16 further comprising randomly selecting a first subset of the algebraof computable features and inputting them into the classifier.
 20. Themethod of claim 19 further comprising randomly selecting a second subsetof features from the first subset of computable features.
 21. The methodof claim 20 further comprising mutating the second subset of features.22. The method of claim 21 further comprising randomly selecting pairsof features from the first subset of computable features.
 23. The methodof claim 22 further comprising applying crossing to each selected pairof features from the first subset of computable features.
 24. The methodof claim 23 further comprising combining the first subset of computablefeatures, the mutated features, and the crossed features to generate athird subset of features.
 25. The method of claim 24 further comprisingselecting a fourth set of features from the algebra of features andadding to the third subset of features.
 26. The method of claim 25further comprising removing duplicate features from the third subset offeatures.
 27. The method of claim 26 further comprising inputting thefeatures from the third subset into the classifier and computingaccuracy of the classifier.
 28. The method of claim 27 furthercomprising ranking the features of the third subset by theircorresponding accuracy.
 29. The method of claim 28 further comprisingbreaking ties, if any, in the ranking based on a total order.
 30. Themethod of claim 29 further comprising maintaining only fittest featuresfrom the third subset to form a fifth subset of the same size as thefirst subset, and replacing the first subset with the fifth subset. 31.The method of claim 30 further comprising repeating the mutating, thecrossing, the removing, the ranking, the selecting, and the maintaininguntil accuracy of a most fit feature converges, wherein the most fitfeature is cached.